1. Field of the Invention
The present invention relates generally to data processing systems and, more specifically, to a computer-implemented method, apparatus, and computer program product for stalling DMA operations during memory migration.
2. Description of the Related Art
According to a trend in computer systems, a system can be reconfigured while running without disrupting data processing. For example, with multiple operating systems running on the computer, while a first one of the operating systems is using a particular block of memory there can arise a need to reallocate the block of memory for use by a second one of the operating systems. In this case, the first operating system must first stop using the block of physical memory before the memory can be reallocated. Or, for example, a problem may be detected in a block of physical memory in which case it may be desirable to remove the memory from operation so that it can be replaced. Once again, whatever operating system was using the block of memory must stop using it.
In certain respects it is relatively straightforward to stop using one block of physical memory and start using another, since mechanisms related to virtual memory management already exist in conventional operating systems to handle aspects of this problem that relate to operating system access to this memory, but these mechanisms depend upon the block of memory being used for only program data that is subject to access by operating systems and that is not subject to access by I/O devices. If the block of memory is subject to access by I/O devices, the problem is more difficult. This access by I/O devices is commonly direct memory access (DMA), although this may not always be the case.
Direct Memory Access (DMA) is the transferring of data from an I/O device to another entity (for example, memory, or another I/O device) without the direct assistance of software. DMA requires gaining control of the I/O bus for the use of the originating I/O device, in order to perform the DMA operation. For shared buses like conventional PCI and PCI-X, control of the I/O bus is obtained by arbitrating for the I/O bus. Turning off the agent that performs the arbitration, i.e. the arbiter, which allows arbitration for the I/O bus, will turn off, i.e. prevent, DMA for all entities that use that I/O bus. For I/O buses like PCI Express where there is only one device per bus, also called a link, access to the bus is obtained by having buffer credits, and stopping DMA involves not giving back any buffer credits to the device at the other end of the bus.
Memory Mapped I/O (MMIO) load without the use of split transactions is the transferring of data from an I/O device to the processor without the target of the operation, i.e. the I/O device that is to supply the data, having to generate its own transaction on the I/O bus to return the requested data. These MMIO operations require the requester, for example, the I/O bridge on behalf of the processor, to continually ask the target (sometimes called polling the target) for the data until the target on one of the polling operations has the data available and returns it to the requester in the requestor's arbitration cycle. The target does not independently access the I/O bus during this operation, and therefore does not need to gain control of the bus. The target returns the data by executing an MMIO load reply operation. The requested data is included in the MMIO load reply.
Memory Mapped I/O (MMIO) load with the use of split transactions is the transferring of data from an I/O device to the processor where the target of the operation, i.e. the I/O device that is to supply the data, may delay passing the data back to the requester. The target must generate its own transaction on the I/O bus to return the requested data. Once the target has the data to be returned, the target generates an MMIO load reply operation on the I/O bus to transfer the data to the requester.
Data that is subject to access by I/O devices may need to be migrated from one physical page to another. When data that is subject to access by I/O devices needs to be migrated, DMA access to that data needs to be suspended during the migration process. The prior art does not offer a solution to migrate this data efficiently, particularly for systems that implement memory mapped I/O (MMIO) using split transactions.
In systems that do not implement split transactions for memory mapped I/O (MMIO) operations, once the requester gains control of the bus, the requester issues its MMIO load request. The requester then maintains control of the operation until the I/O device replies to the MMIO load request with the requested data. In this manner, the target is not required to gain control of the bus.
In systems that implement split transactions, the requester issues its MMIO load request once the requester gains control of the bus. Once the requester issues its MMIO load request, the requester relinquishes control of the bus. The target I/O device, once it has the requested data, then must gain control of the bus in order to reply to the request and send its data to the requester.
If DMA is stopped by turning off the access to the bus by an I/O device, in systems that use split transactions, the portion system that uses the stalled portion of the I/O system, will stall. This is because an I/O device that needs to gain control of the bus in order to execute an MMIO load reply will never be able to gain control of the bus. If the I/O device cannot gain control of the bus and, thereby, cannot execute its MMIO load reply, that MMIO load operation will never be able to complete, thereby stalling the processor issuing the MMIO load operation.
Furthermore, if the DMA is stopped closer to the processor rather than immediately adjacent to the I/O device, such that the I/O device can still access the bus, then DMA write requests that get stalled can prevent the MMIO load split response from returning to the processor. The reason for this is that PCI operation ordering rules require that an MMIO load reply cannot be returned to the processor doing the load operation until all previous DMA write operations have been completed from the same I/O device. Thus, MMIO load replies can get stuck in a queue behind previous DMA write requests that are waiting to be processed, but which cannot be processed due to the memory migration operation.